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Developing  integrated  test  concepts  for  aircraft  requires  close  interaction  between  contractors, 
acquisition  officials ,  system  users,  U.S.  Air  Force  test  agencies,  and  the  Federal  Aviation 
Administration.  Integrated  test  emphasis  within  the  Department  of  Defense  arrived  at  a 
prominent  time  to  make  a  positive  impact  on  the  C-130J,  KC-X,  E-3  DRAGON,1  and  Joint 
Cargo  Aircraft  test  programs.  Mobility  and  surveillance  airframes  are  exceeding  corrosion  and 
fatigue  models  based  on  greater  flight  rates ,  requiring  new  assets  more  swiftly.  All  stakeholders 
in  the  programs  have  a  vested  interest  in  making  the  test  and  evaluation  program  as  efficient  as 
possible.  This  article  reviews  Department  of  Defense  integrated  test  concept,  identifies 
operational  test  characteristics  required  by  public  law,  and  discusses  integrated  test  methods, 
which  comply  with  law  and  policy.  Implementation  challenges  are  also  discussed,  including 
mobility  and  surveillance  aircraft  test  community  methods  addressing  integrated  testing 
challenges,  tracking  test  events,  and  identifying  integrated  test  opportunities. 

Key  words:  Acquisition  team;  aircraft  test;  collaboration;  contractor  test;  developmental 
test;  FAA  certification;  law  and  policy;  operational  test;  weapons  systems. 


Operational  Test  and  Evaluation 

(OT&E)  ensures  the  warfighter  is 

provided  with  suitable  and  effective 
weapon  systems.  To  rapidly  provide 
new  or  modified  systems  for  our 
airmen,  the  acquisition  process  requires  greater  effi¬ 
ciency.  Increasing  T&E  effectiveness  can  assist  this 
effort.  Integrated  weapon  system  testing  combines 
Contractor  Test  (CT),  Developmental  Test  (DT),  and 
Operational  Test  (OT).  Integrated  tests  have  the 
potential  to  shorten  acquisition  timelines,  while 

balancing  cost,  performance,  and  schedule  require¬ 

ments.  DT  and  OT  terms  used  in  this  article  are  broad 
statements  for  a  type  of  T&E,  not  terms  used  to 
describe  testing  controlled  by  any  one  particular 
organization. 

In  a  Joint  memorandum,  dated  25  April  2008,  the 
Director  of  Operational  Test  and  Evaluation  and 
Deputy  Undersecretary  of  Defense  for  Acquisition  and 
Technology  defined  Integrated  Testing  (IT)  as  “The 
collaborative  planning  and  execution  of  test  phases  and 
events  to  provide  shared  data  in  support  of  indepen¬ 
dent  analysis,  evaluation  and  reporting  by  all  stake¬ 
holders  particularly  the  DT  (both  government  and 
contractor)  and  OT  communities”  (McQueary  and 
Finley  2008). 


IT  is  more  than  combining  developmental  and 
operational  testing.  IT  is  establishing  a  test  process 
that  fully  synergizes  all  stakeholder  objectives  into  a 
single  continuum  with  the  goal  of  achieving  the  most 
effective,  efficient  test  program  possible.  The  scope 
includes  CT,  DT,  OT,  and,  in  the  case  of  the  KC-X 
and  Joint  Cargo  Aircraft  QCA)  programs,  the  Federal 
Aviation  Administration  (FAA)  certification  activities. 
Today,  DT  involves  both  government  and  contractor 
test  organizations,  ensuring  weapon  system  specifica¬ 
tions  are  met.  However,  some  data  produced  by  DT 
and  properly  conducted  test  events  satisfying  U.S. 
Code,  Title  10,  Section  2399,  “Operational  Test  and 
Evaluation  of  Defense  Acquisition  Programs”  can 
support  operational  T&E  needs.  Integrated  DT  and 
OT  planning  and  execution  can  reduce  total  test 
expenditures  and  timelines. 

What  does  this  mean  for  all  stakeholders  in  an 
acquisition  program?  Data  from  an  IT  can  be  used  by  the 
contractor  for  design  improvements,  by  developmental 
evaluators  for  risk  assessments,  and  by  the  operational 
evaluators  for  operational  assessments.  Early  and  open 
communication  between  all  parties  within  the  acquisi¬ 
tion  team  is  a  must!  The  user,  program  office,  contractor, 
DT,  and  OT  testers  must  be  active  early  in  a  program 
before  a  request  for  proposal  is  delivered  to  potential 
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contractors.  IT  can  better  use  limited  resources  such  as 
test  assets  and  ranges,  eliminate  redundant  test  events, 
and  reduce  the  overall  program  cost  and  schedule.  The 
Department  of  Defense  (DoD)  recognizes  the  potential 
benefits  and  has  directed  implementation  of  the 
integrated  test  concept. 

DoD  integrated  test  concept 

Integrated  testing  is  not  a  new  concept  within  the 
DoD.  The  concept  existed  in  the  May  2003, 
Department  of  Defense  Instruction  (DoDI)  5000.2, 
Operation  of  the  Defense  Acquisition  System.  How¬ 
ever,  the  policy  was  neither  well  defined  nor  consis¬ 
tently  implemented  in  the  acquisition  process.  The 
DoD  further  refined  integrated  test  direction  with  the 
revised  DoDI  5000.2  in  December  2008.  The  new 
instruction  refines  integrated  test  concepts  to  include 
the  following  direction  (emphasis  and  acronym  defi¬ 
nitions  added): 

“EMD  [Engineering  Manufacturing  and  De¬ 
velopment]  effectively  integrates  the  acquisition, 
engineering,  and  manufacturing  development 
processes  with  T&E  [Test  and  Evaluation]  (see 
Enclosure  6).  T&E  shall  be  conducted  in  an 
appropriate  continuum  of  live,  virtual,  and 
constructive  system  and  operational  environ¬ 
ments.  Developmental  and  operational  test 
activities  shall  be  integrated  and  seamless 
throughout  the  phase.  Evaluations  shall  take 
into  account  all  available  and  relevant  data  and 
information  from  contractor  and  government 
sources”  (DoD  2008,  24). 

Additionally,  the  new  DoDI  directs  “Integrated 
Test”  in  the  following  manner  (acronym  definitions 
added): 

“ The  PM  [Program  Manager],  in  concert  with 
the  user  and  the  T&E  community,  shall 
coordinate  DT&E,  OT&E,  LFT&E  [Live 
Fire  Test  and  Evaluation],  family-of-sy stems 
interoperability  testing,  information  assurance 
testing,  and  modeling  and  simulation  (M&S) 
activities  into  an  efficient  continuum,  closely 
integrated  with  requirements  definition  and 
systems  design  and  development.  The  T&E 
strategy  shall  provide  information  about  risk 
and  risk  mitigation,  provide  empirical  data  to 
validate  models  and  simulations,  evaluate  tech¬ 
nical  peformance  and  system  maturity,  and 
determine  whether  systems  are  operationally 
effective,  suitable,  and  survivable  against  the 
threat  detailed  in  the  STAR  [System  Threat 


Assessment  Report]  or  STA  [System  Threat 
Assessment],  The  T&E  strategy  shall  also  address 
development  and  assessment  of  the  weapons 
support  equipment  during  the  EMD  Phase, 
and  into  production,  to  ensure  satisfactory  test 
system  measurement  performance,  calibration 
traceability  and  support,  required  diagnostics, 
and  safety.  Adequate  time  and  resources  shall  be 
planned  to  support  pre-test  predictions  and  post¬ 
test  reconciliation  of  models  and  test  results,  for 
all  major  test  events”  ( DoD  2008,  50). 

The  U.S.  Air  Force  Operational  Test  and  Evaluation 
Center  (AFOTEC)  Director  of  Operations  released  a 
memorandum  on  17  October  2008  to  operational  test 
planners  stating:  “As  testers,  we  can  affect  the  cost- 
schedule-performance  problem  facing  program  offices  by 
making  better  use  of  limited  test  assets  and  test  ranges  to 
eliminate  unnecessary  overlap  of  test  events,  better  assure 
that  systems  are  ready  for  OT,  and  reduce  the  overall 
time  required  for  testing”  (Eck  2008). 

The  AFOTEC  commander  has  taken  the  integrated 
test  concept  one  step  further,  directing  OT  plans  to 
incorporate  integrated  events.  He  has  directed  OT 
planners  to  develop  plans  and  concepts  earlier  to 
facilitate  integrated  test  planning.  Figure  1  shows  the 
OT  early  planning  concept.  At  Milestone  A,  or  key 
decision  point,  the  OT  planners  provide  an  initial  OT 
design  to  support  development  of  IT  in  the  T&E 
strategy.  At  Milestone  B,  the  OT  design  is  expanded  as 
the  system  capabilities  are  further  developed  and 
defined.  Between  Milestone  B  and  C,  the  OT  design 
becomes  a  detailed  test  plan  and  is  documented  in  an 
Integrated  Operational  Test  and  Evaluation  (OT&E) 
plan.  Throughout  the  acquisition  process,  the  avail¬ 
ability  of  the  OT  design  is  used  to  influence  contractor 
and  DT  planning  and  to  support  IT.  After  the 
AFOTEC  commander  approves  the  OT&E  plan  with 
integrated  events,  the  plan  will  be  sent  to  the  Program 
Executive  Officer  (PEO)  for  acknowledgment  regard¬ 
ing  OT  use  of  the  integrated  events  to  ensure  their 
execution.  The  PEO  has  a  vested  interest  in  ensuring 
efficient  test  execution. 

Combining  DoD  and  AFOTEC  commander’s 
direction  together,  the  key  characteristics  required  for 
integrated  test  are  as  follows: 

1.  collaborative  planning  and  execution  of  test 
events; 

2.  shared  data; 

3.  stakeholder  independent  data  analysis,  evalua¬ 
tion,  and  reporting; 

4.  establishing  risks  and  risk  mitigation  to  decision 
makers  in  a  timely  manner; 
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Figure  1.  Earlier  operational  testing  planning  concept. 


5.  efficient  use  of  limited  assets,  reducing  overall 
time  required  for  testing; 

6.  elimination  of  unnecessary  overlap  of  test  events; 
and 

7.  documentation  in  an  OT&E  plan  with  integrated 
events  signed  by  the  OT  commander  and 
acknowledged  by  the  PEO. 


involved.  The  contractor  is  quite  clearly  engaged  in 
most  DT  and  by  definition  in  any  CT.  However,  there 
are  methods  available  to  ensure  the  operational  testers 
conduct  the  test  event,  even  though  during  other  test 
events  during  the  test  period  the  contractor  is  involved. 
Title  10,  Section  139,  “Director  of  Operational  Test 
and  Evaluation  (DOT&E)”  defines  operational  test 
and  evaluation  as  follows: 


Operational  test  public  law  limitations 

Blending  the  direction  of  senior  DoD  leaders  with 
public  law  governing  OT  is  challenging.  Title  10, 
section  2399,  Operational  Test  and  Evaluation  of 
Defense  Acquisition  Programs,  contains  clauses  that 
make  IT  difficult;  in  particular, 

“(d)  Impartiality  of  Contractor  Testing  Person¬ 
nel  —  In  the  case  of  a  major  defense  acquisition 
program  (as  defined  in  subsection  (a)(2)),  no 
person  employed  by  the  contractor  for  the  system 
being  tested  may  be  involved  in  the  conduct  of  the 
operational  test  and  evaluation  required  under 
subsection  (a).  The  limitation  in  the  preceding 
sentence  does  not  apply  to  the  extent  that  the 
Secretary  of  Defense  plans  for  persons  employed 
by  that  contractor  to  be  involved  in  the  operation, 
maintenance,  and  support  of  the  system  being 
tested  when  the  system  is  deployed  in  combat"  (10 
USC  2399). 

For  IT,  claiming  “credit”  regarding  an  event  for  OT 
purposes  becomes  a  challenge  when  contractors  are 


“(A)  The  term  “operational  test  and  evaluation” 
means  -  (i)  the  field  test,  under  realistic  combat 
conditions,  of  any  item  of  (or  key  component  of 
weapons,  equipment,  or  munitions  for  the 
purpose  of  determining  the  effectiveness  and 
suitability  of  the  weapons,  equipment,  or  muni¬ 
tions  for  use  in  combat  by  typical  military  users; 
and  (ii)  the  evaluation  of  the  results  of  such  test” 

(10  USC  139). 

The  key  words  here  are  “field  test,  under  realistic 
combat  conditions”  and  by  “typical  military  users.” 
There  is  no  clear  definition  what  the  terms  mean,  so 
operational  testers  must  exercise  reasonable  judg¬ 
ment.  Generally,  contractor  and  developmental  test 
conditions  are  not  realistic  combat  conditions  by 
design.  Contractor  and  developmental  testing  is 
conducted  under  controlled  conditions  designed  to 
evaluate  the  system,  subsystem,  or  function,  which  is 
the  focus  of  the  test.  The  uncontrolled  conditions 
inherent  in  realistic  combat  conditions  conflict  with  a 
normal  structured  test  approach.  Again,  there  are 
times  and  places  where  conducting  a  test  event  under 
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realistic  combat  conditions  with  typical  military 
users  can  meet  contractor  and  DT  needs  and  fulfill 
OT  needs. 

DoDI  5000.2  further  requires  the  following  from 
OT&E: 

“ The  independent  planning  of  dedicated  IOT&E 
(i.e.,  the  OT&E  required  by  paragraphs  (a)  and 
(b)  of  section  2399  of  Reference  (k)),  and 
Follow-on  OT&E  (FOT&E),  if  required \  shall 
be  the  responsibility  of  the  appropriate  OT  agency 
(OTA).  Evaluations  shall  include  a  comparison 
with  current  mission  capabilities  using  existing 
data ,  so  that  measurable  improvements  can  be 
determined.  If  such  evaluation  is  considered  costly 
relative  to  the  benefits  gained,  the  PM  shall 
propose  an  alternative  evaluation  approach.  This 
evaluation  shall  make  a  clear  distinction  between 
deficiencies  uncovered  during  testing  relative  to 
the  approved  requirements  and  recommendations 
for  improvement  not  directly  linked  to  require¬ 
ments.  A  DOT&E-approved  LFT&E  strategy 
shall  guide  LFT&E  activity”  (DoD  2008,  24). 

“. During  OT&E,  a  clear  distinction  shall  be 
made  between  performance  values  that  do  not 
meet  threshold  requirements  in  the  user  capabil¬ 
ities  document  and  performance  values  that 
should  be  improved  to  provide  enhanced  opera¬ 
tional  capability  in  future  upgrades"  (DoD 
2008,  50). 

“The  Department  of  Defense  may  not  conduct 
OT&E,  including  operational  assessment  (OA), 
IOT&E,  or  FOT&E,  until  the  DOT&E 
approves,  in  writing,  the  OT&E  portions  of  the 
T&E  plan  for  programs  on  the  Office  of  the 
Secretary  of  Defense  (OSD)  T&E  Oversight  List 
and  the  adequacy  of  the  plans  (including  the 
projected  level  of  funding)  for  the  OT&E  to  be 
conducted  in  connection  with  that  program.  This 
does  not  preclude  the  use  of  data  from  other  test 
events  in  OT&E  evaluations.  OTA  and  DOT&E 
evaluators  shall  take  into  account  all  available  and 
relevant  data  and  information  from  contractor  and 
government  sources”  (DoD  2008,  25). 

The  duty  of  the  integrated  test  planner  is  to  blend 
DT,  often  conducted  with  the  contractor,  together 
with  OT,  while  complying  with  public  law  and  senior 
DoD  leadership  direction.  Additionally,  the  OT 
planner  must  balance  policy  guidance  and  design  a 
highly  efficient  T&E  framework  to  meet  shorter 
schedule  and  smaller  budget  constraints. 


Overcoming  operational  test  public 
law  limitations 

Public  law  directs  that  operational  tests  employ 
production  representative  test  articles  and  are  conduct¬ 
ed  under  realistic  combat  conditions  by  typical  military 
users.  These  directives  appear  to  conflict  with  the  DoD 
guidance  regarding  IT.  In  reality,  conducting  aircraft 
IT  events  for  OT  “credit”  is  not  difficult.  Meeting  the 
“typical  military  user”  and  “realistic  combat  conditions” 
requires  coordination  with  the  developmental  testers. 
Arranging  a  particular  test  event  to  have  an  OT  crew, 
in  most  cases,  is  a  simple  aircrew  scheduling  problem. 
Flying  the  test  event  under  realistic  combat  conditions 
can  also  be  accomplished  without  significant  effort. 

Most  aircraft  test  teams  include  both  an  operational 
and  developmental  test  contingent.  DT  normally 
follows  a  “build-up”  approach,  where  early  test  points 
are  designed  or  constrained  to  evaluate  specific 
conditions.  Later,  test  points  generally  are  less 
constrained  and  are  perfect  opportunities  to  implement 
an  integrated  test  event  counting  for  operational  test 
credit.  Planning  for  the  OT  team  contingent  to 
conduct  these  less  stringent  test  points  under  opera¬ 
tional  conditions  allows  a  corresponding  reduction  in 
the  separate  OT  events.  The  test  planners  integrate 
evaluation  requirements  to  effectively  accomplish  the 
test  event,  while  providing  data  for  each  stakeholder. 
The  test  planners  attempt  to  ensure  that  test  events 
accomplished  in  DT  are  not  replicated  in  OT. 

For  example,  in  recent  C-130  J  aircraft  defensive 
system  evaluation,  DT  and  OT  evaluators  were  able  to 
integrate  several  test  events.  During  DT  of  a  radar 
warning  receiver,  the  last  hour  of  range  time  was  used 
by  OT,  flying  operational  scenarios  against  the  same 
systems  used  by  DT  earlier  in  the  flight.  By  conducting 
the  test  events  in  this  manner,  testers  reduced  the 
acquisition  program’s  expensive  range  costs,  test  flight 
hours,  schedule,  and  total  number  of  test  sorties 
required.  Additionally,  the  DT  team  gained  an 
understanding  of  the  subsystem  performance  when 
combined  with  operational  tactics  and  procedures. 
Some  key  system  characteristics  were  identified  during 
the  test  planning,  which  facilitated  the  integrated  test 
planning.  First,  the  system  development  was  stable, 
and  the  risk  of  major  changes  to  the  hardware  or 
software  were  minimal  (i.e.,  the  system  could  reason¬ 
ably  be  called  “production  representative”  for  public 
law  OT  purposes).  Second,  the  cost  and  schedule 
difficulties  associated  with  the  range  time  and  aircraft 
sorties  were  identified  as  areas  where  IT  could 
significantly  impact  the  overall  acquisition  program 
cost  and  timeline. 

Even  when  specific  DT  events  fail  to  meet  public 
law  criteria  for  OT,  the  events  can  still  positively 
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impact  OT  events  and  thus  the  program’s  cost  and 
schedule.  Unless  specified  in  a  requirements  document, 
the  confidence  needed  in  a  specific  performance 
requirement  is  an  OT  subjective  determination. 
Operational  evaluators  have  the  planning  latitude  to 
address  areas  where  operational  risk  or  impact  appears 
greatest  or  reduce  the  number  of  test  events  required 
where  the  operational  risk  or  impact  appears  low. 
Using  test  event  data  from  DT  to  assess  risks  can 
reduce  OT  cost  and  schedule.  For  example,  if  DT 
confirms  an  aircraft’s  precision  approach  and  landing 
capability  through  numerous  test  events  to  satisfy  FAA 
requirements,  the  operational  planner  could  assume  the 
system  performance  operational  risk  is  low  and 
significantly  reduce  the  total  number  of  OT  events 
evaluating  precision  approach  and  landing  capabilities. 
As  straight  forward  as  the  concept  is,  the  operational 
evaluators  must  address  several  IT  implementation 
hurdles  before  securing  shared  data,  especially  for  the 
KC-X  program. 

Integrated  test  implementation 
challenges 

The  KC-X  integrated  test  planning  effort  faces 
various  barriers;  foremost  is  the  contracting  process. 
The  current  acquisition  strategy  identifies  using  a 
commercial  derivative  aircraft  complete  with  FAA 
certifications.2  Achieving  appropriate  certifications  is 
left  to  each  bidding  contractor  and  is  an  area  of 
competition  for  determining  the  eventual  contract 
winner.  In  order  to  support  IT  prior  to  source 
selection,  each  contractor  would  need  to  interface  with 
the  complete  test  community  to  develop  an  integrated 
test  and  then  include  the  developed  plan  as  part  of  their 
proposal,  a  significant  concern  for  most  contracting 
officers.  Without  some  alternative  method,  developing 
an  executable  integrated  test  requires  coordination  with 
the  winning  contractor  after  source  selection,  when 
changing  the  contract  to  accommodate  specific  IT 
events  may  cause  expensive  contract  modifications. 

The  FAA  aircraft  certification  process  differs 
significantly  from  the  Air  Force  (AF)  process.  In 
particular,  without  specific  wording  in  the  contract, 
data  submitted  to  the  FAA  for  certification  will  not  be 
available  to  the  AF  test  community.  The  approach 
conflicts  with  the  shared  data  characteristic  outlined  in 
DoD  directives.  Since  the  acquisition  strategy  currently 
calls  for  a  commercial  derivative  aircraft,  test  data  and 
methods  are  competitive  factors  among  contractors 
(not  only  for  the  KC-X  contract  but  also  in  all  their 
business  ventures)  and  will  likely  be  a  discussion  item 
within  the  contract  negotiations.  Additionally,  the 
FAA  is  the  approval  authority  for  the  certification 
process,  and  integrating  the  FAA  test  events  with 
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Figure  2.  Conceptual  test  planning  timelines. 


other  test  events  as  well  as  obtaining  required  approvals 
from  all  stakeholders  poses  further  challenges. 

Developmental  and  operational  test  planning  timing 
issues  also  challenge  implementing  IT.  Figure  2 
illustrates  the  nominal  timing  issues  and  shows  where 
test  events  are  normally  created.  The  figure  also 
illustrates  where  test  planners  need  to  know  all  the 
events  to  integrate  them  effectively.  Currently,  test¬ 
planning  timelines  are  not  arranged  to  facilitate 
integrated  test  planning.  Normally,  CT  plans  are 
produced  before  DT  plans,  which  are  produced  before 
OT  plans.  To  completely  chart  an  IT  in  one  iteration 
requires  contractor,  developmental,  and  operational 
test  planners  working  simultaneously,  communicating 
and  integrating  test  events  among  themselves.  The 
DoD  recognizes  the  need  and  specifically  calls  for 
collaborative  test  development  as  an  IT  characteristic. 
To  perform  collaborative  planning,  developmental  and 
operational  planners  are  required  earlier  in  the  test 
planning  process.  How  early  depends  upon  the  specific 
program. 

Additionally,  implementation  of  an  integrated  test 
plan  is  challenging.  Perhaps  the  program’s  Test  and 
Evaluation  Master  Plan  (TEMP)  should  become  the 
repository  of  integrated  test  events.  Establishing  the 
TEMP  as  the  repository  of  integrated  test  events 
permits  the  visibility  of  the  required  events  and  allows 
detailed  test  planning  to  mature  with  the  program.  The 
TEMP  supporting  milestone  B  (the  decision  point 
within  the  DoD  acquisition  system  to  begin  system 
engineering  and  demonstration)  would  have  detailed 
CT  information,  general  DT  detail,  and  conceptual 
OT  detail.  At  milestone  C  (the  decision  point  within 
the  DoD  acquisition  system  to  begin  production  and 
deployment),  the  TEMP  could  be  revised,  adding 
details  to  the  developmental  and  operational  test 
sections.  Because  the  TEMP  would  include  most  test 
events,  execution  of  events,  and  test  cost,  the  schedule 
optimization  should  naturally  occur  through  the 
TEMP  review  processes  already  in  place.  The  totality 
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of  the  test  events  should  be  evident,  clearly  identifying 
which  events  support  both  DT  and  OT.  A  method  to 
efficiently  handle  changes  is  required  and  is  beyond  the 
scope  of  this  article.  Both  test  planners  and  senior 
DoD  officials  should  be  able  to  capitalize  upon  the 
visibility,  combining  OT  planning  with  DT  test 
events,  and  optimize  test  expenditure  and  schedule. 

Figure  3  illustrates  the  source  documents  for  key  test 
events  for  a  program.  OT  plans  are  developed  from 
concepts  of  employment  and  required  capabilities 
documents.  DT  plans  are  derived  from  required 
capabilities  documents  and  system  specifications.  For 
most  programs,  there  is  not  a  single  DT  plan.  Instead, 
different  DT  agencies  develop  test  plans  addressing  the 
test  events  they  are  chartered  to  test.  So,  while  shown 
here  as  a  single  document,  there  may  be  many  DT  test 
plans.  Contractor  test  plans  are  developed  from  system 
specifications,  contractual  requirements,  and  in  this 
case,  FAA  certification  requirements.  Each  of  the 
documents  is  related  and  may  contain  the  same 
requirement  specified  in  different  ways.  Development 
of  the  OT,  DT,  and  CT  plans  separately  increases  the 
probability  that  the  multiple  plans  may  contain 
duplicative  test  events.  A  method  of  integrating  all  of 
the  events  derived  from  the  source  of  the  requirement, 
where  each  organization  can  see  the  test  events  in 
totality,  is  required.  Establishing  informative  methods 
to  reduce  duplication  is  the  ultimate  goal  of  IT.  The 
KC-X,  JCA,  and  E-3  DRAGON  programs  are 
pursuing  the  integrated  concept  by  identifying  test 
events  early,  combining  them,  and  minimizing  the  test 
footprint,  with  the  goal  to  field  aircraft  systems  more 
rapidly. 

Current  integrated  test  planning 
approaches 

The  KC-X  test  community  includes  developmental 
and  operational  testers,  FAA,  program  office,  and 


Figure  4.  Integrated  test  approach  conceptual  diagram. 


DOT&E  personnel.  The  test  community  established 
an  Integrated  Test  Team  (ITT),  which  meets  regularly. 
At  the  meetings,  the  team  defines  required  information 
to  develop  integrated  test  events  and  creates  a  process 
to  capture  and  communicate  those  events. 

The  first  challenge  the  KC-X  ITT  overcame  was 
development  of  a  conceptual  framework  to  plan  IT. 
After  numerous  discussions,  the  ITT  decided  the  best 
IT  implementation  approach  started  with  the  source 
documents  shown  in  Figure  3.  Rather  than  employing 
the  source  documents  individually  to  create  separate 
contractor,  DT,  and  OT  plans,  the  ITT  decided  to 
collect  all  the  test  events  together  into  an  Integrated 
Test  Events  Matrix  (ITEM)  as  a  document  that  can 
highlight  test  duplication  and  redundancy.  Figure  4 
illustrates  the  concept.  The  ITEM  could  become  a 
source  document  for  TEMP  development  or  perhaps 
be  included  within  the  TEMP.  With  the  ITEM  and 
TEMP,  the  respective  test  organizations  can  develop 
their  individual  test  plans  and  include  them  with  the 
TEMP  as  attachments.  Additionally,  the  program 
office  can  use  the  ITEM  to  communicate  to  potential 
contractors  the  desired  nature  of  the  integrated  test. 
The  process  could  have  the  TEMP  functioning  as  the 
ITEM  repository,  and  individual  test  plans  would 
support  execution  of  the  IT  described  in  the  TEMP. 

To  affect  KC-X  IT,  the  team  concluded  that  each 
test  event  needs  to  be  described  in  terms  of  the 
following  four  descriptive  tables: 

•  Table  T.  Source  -  Requirements  that  drive  test 
event  or  activity  needs.  Examples  include  the 
Capabilities  Development  Document  (CDD),  Sys¬ 
tem  Requirements  Document  (SRD),  Concept  of 
Employment  (CONEMP),  Statement  of  Objec¬ 
tives  (SOO),  or  civil  certification  plan  requirements. 

•  Table  2\  Description  -  A  definition  of  the  system, 
subsystem,  or  function  evaluated,  highlighting 
the  objectives,  participating  test  organizations, 
test  methodology,  measures,  metrics,  and  perfor¬ 
mance  criteria.  Included  are  the  test  event- 
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Table  1.  Source  documentation  driving  test  requirements. 

Source 

Describe  which  requirement(s)  dictate  the  particular  test  to  be  conducted  (e.g.,  RFP,  certification  ph 

other  requirement  source  paragraph) 

in,  or 

CDD 

SRD 

Supportability 

requirements 

CONOPS/ 

Certifications  CONEMPS 

SOO 

CDD  language 
associated  with 
the  particular  test, 
if  applicable 

SRD  language 

associated  with  the 
particular  test,  if 
applicable 

Supportability  requirements 
that  are  the  basis  for  the 
particular  test,  if 
applicable 

Certification  that  CONOP/CONEMPs 
dictates  the  that  dictate  the 

particular  test,  if  particular  test,  if 

applicable  applicable 

SOO  language  that 
dictates  the 
particular  test,  if 
applicable 

CDD,  Capabilities  Development  Document;  SRD,  Systems  Requirements  Document;  CONOPS,  concept  of  operations;  CONEMPS,  concept 
of  employment;  SOO,  statement  of  objectives. 


dependent  thresholds  to  verify  a  successful  test 
completion  and  the  method  to  complete  evaluation 
(not  shown),  such  as  inspection,  analysis,  ground 
or  flight  demonstrations  or  laboratory  tests. 

•  Table  3:  Conditions  -  The  system  configuration, 
test  environment,  and  personnel  essential  to 
conduct  the  test.  The  condition  section  is  key  in 
determining  if  combined  testing  is  possible.  Here 
is  where  the  test  planner  can  begin  to  see 
different  situations  separating  contractor,  DT, 
or  OT  planning.  Some  tests  may  be  integrated,  as 
permitted  by  law  and  DoD  direction,  and  are 
continued  into  the  resources  section. 

•  Table  4\  Resources  -  Test  articles,  support 
equipment,  simulations,  models,  facilities,  and 
personnel  required.  The  Resource  portion  can 
also  identify  resources  that  can  be  shared  once 
integrated  testing  is  deemed  valid.  For  instance, 
contractor,  developmental,  and  operational  testers 
can  share  aircraft  generation  equipment  if  the 
equipment  is  listed  early  in  the  program. 
Duplicated  resources  can  be  highlighted  and 


reduced.  Further,  test  range  activities,  one  of 
the  more  expensive  test  resources,  may  be 
combined,  permitting  more  programs  earlier 
opportunities  for  evaluation. 

The  C-130J  program  has  taken  another  approach. 
Two  different  follow-on  operational  tests  have  inte¬ 
grated  developmental  and  operational  events.  The  C- 
130J  ALR-56M  radar  warning  receiver  tests  required 
range  support  for  both  DT  and  OT.  Range  costs  and 
setup  times  were  recognized  by  both  communities  as 
likely  integrated  test  cost-savings  areas.  The  testers 
combined  OT  with  the  DT  mission  by  loading  more 
chaff  and  performing  some  OT  events  at  the  end  of 
each  DT  mission.  OT  “purchased”  the  last  hour  on  the 
range  for  numerous  tests  at  higher/lower  altitudes, 
airspace  and  terrain  masking  events  highlighting  more 
operationally  realistic  activities.  The  concept  required 
early  test  planning  for  both  entities.  The  integrated  test 
concept  gave  OT  more  data  and  made  the  best  use  of 
range  time  plus  maintenance  support  and  other  flight- 
related  assets.  Further,  OT  observed  early  DT 


Table  2.  Overall  test  event  description. 


Test  Description 

Describe  each  test  to  be  conducted.  Include,  as  a  minimum  the  system,  subsystem  or  function  to  be  tested,  the  objectives  for 
the  particular  test,  participating  test  organizations,  the  applicable  testing  methodology,  measures  and  metrics,  and 
the  acceptable  range/threshold,  for  each  of  the  metrics,  that  indicates  a  successful  test  completion. 

Subsystem/ 

function 

Test 

objectives 

Test 

organization 

Test 

methodology 

Measures 
and  metrics 

Acceptable 
range/ threshold 

Primary  subsystem/ 
function  to  be 
tested  (include 
related  systems 
in  parentheses 
if  desired). 

Primary  objective  or 
purpose  of  the 
particular  test 
(secondary 
objective  can 
be  included  in 
parentheses). 

RTO  for  the 
particular  test 
(PTOs  can  be 
listed  in 
parentheses). 

The  desired  testing 
methodology  for 
this  particular  test 
(secondary,  tertiary, 
etc.  methodologies 
should  be  listed  in 
parentheses  and  in 
order  of  preference). 

The  one  primary 
metric  to  be  used 
to  evaluate  test 
results  (additional 
metrics  can  be 
included  in 
parentheses). 

The  success  threshold  and/or 
range  of  results  for  the 
primary  metric  to  be  used  for 
this  test  (additional 
thresholds  and/or  ranges  can 
be  listed  in  parentheses  as  a 
one-to-one  match  for  the 
metrics/ measurements  listed 
in  the  preceding  column). 

RTO,  responsible  test  organization;  PTO,  participating  test  organization. 
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Table  3.  Test  event  conditions. 


Conditions 


Describe  the  conditions  under  which  the  test  shall  be  conducted.  Include,  as  a  minimum,  the  testing  configuration,  the  testing 
environment,  and  any  desired  test  crew  or  test  conductors  in  addition  to  those  provided  by  the  contractor. 

Configuration 

Environment 

Combined  testing?  (Y/N)  Crew/Tester 

The  physical  configuration  for  the 
object  to  be  configured  to 
conduct  the  test. 

The  desired  physical  environment 
for  the  particular  test  (e.g., 
standard  day,  no  wind). 

Indicate  if  more  than  one  testing  The  primary  crew  and/or  test 
organization  can  be  involved  conductor  for  the  particular  test 

with  the  particular  test.  (additional  crew/testers  can  be 

listed  in  parentheses). 

assessments  into  the  effectiveness  and  suitability  of  the 
ALR-56M  defensive  system,  fulfilling  OT-dedicated 
testing.  C-130J  testing  continues  today  with  a 
formation  station  keeping  equipment  evaluations.  Both 
communities  are  repeating  the  integrated  planning  and 
execution  process  as  DT  and  OT  flights  accomplish 
similar  assessments.  For  example,  the  OT  community 
is  planning  to  use  data  from  DT  flights  that  chase 
instrument  meteorological  conditions  as  part  of  the 
OT  test  data.  Integrating  these  tests  allows  the 
program  to  evaluate  very  specific  events  once  without 
having  to  repeat  these  tests  during  OT.  Having  the 
combined  DT/OT  events  permits  OT  to  reduce  the 
overall  test  footprint,  fulfilling  aircraft  verification  and 
validation  more  swiftly. 

The  E-3  DRAGON  program  adds  another  layer  to 
the  KC-X  ITEM  concept  with  requirements  integrating 
tests  with  NATO  Airborne  Warning  and  Control 
System  (AWACS).  One  additional  section  lists  com¬ 
monalities  between  the  U.S.  and  NATO  aircraft,  while 
highlighting  NATO’s  proximity  to  the  Southwest  Asia 
and  Arctic  theaters;  whereas  the  U.S.  is  nearer  to  the 
international  dateline,  Eastern  Asia  and  Australia.  If  the 
commonalities  between  U.S.  and  NATO  aircraft  are 
sufficient,  DT  and  OT  could  be  divided  and  information 
shared,  reducing  overall  flight-hour  costs  and  schedule. 

Table  4.  Test  event  resources. 


The  KC-X  program  has  created  new  IT  approaches. 
Bidding  contractors  are  mandated  to  conduct  cold/hot 
weather  testing  and  also  include  IT  by  requesting  that 
DT  ensure  verification  of  subsystem  operations  and  by 
requesting  OT  conduct  a  preflight,  ensuring  that  crews 
in  cold/hot  environments  can  successfully  perform  a 
mission.  Further,  the  KC-X  program  has  requested 
type-I  contractor  training  to  be  conducted  concurrently 
with  mid-phase  EMD  testing.  For  instance,  OT  crews 
will  be  certified  flying  the  KC-X  EMD  aircraft  before 
MS-C  and  maintain  currency  by  conducting  some 
take-offs  and  landings  on  a  typical  DT  mission 
planned  for  a  range  evaluation.  Taking  IT  one  step 
further,  OT  can  conduct  radio  evaluations  to/from  a 
planned  DT  test  range  if  the  sortie  length  between 
takeoff  and  range  entry  permits. 

KC-X  OT  crews  will  participate  in  receiver  aircraft 
certifications,  both  on  the  KC-X  as  a  tanker  and 
various  multi-seat  aircraft  receivers.  DT  and  OT  events 
are  combined  before  the  first  government  air  refueling 
flights  reviewing  KC-X  suitability  and  effectiveness  as 
early  as  possible.  DT-only  missions  will  cover  the  first 
couple  of  air  refueling  sorties,  demonstrating  safety  and 
air  refueling  system  maturity.  If  no  major  air  refueling 
subsystem  warrants  change,  such  as  boom  flight 
controls,  then  OT  can  jointly  fly  on  following  sorties 


Resources 


Describe  the  physical  resources  required  for  successful  test  completion.  Include,  as  a  minimum,  articles  required  for  the  test, 
the  supporting  assets  (e.g.,  chase  aircraft  and  so  on)  needed  and  the  simulations,  models,  test  beds,  facilities,  and 
other  personnel  besides  the  primary  testing  crew  or  test  conductors. 


Test  articles 

Supporting  assets 

Simulations,  models, 
and  test  beds 

Facilities 

Personnel 

Special 

List  articles 

List  equipment  or 

List  simulation,  model, 

List  primary  testing 

List  personnel  other 

List  resources  the 

required  for  the 

other  physical  assets 

or  test  bed  needed  for 

facility  to  conduct 

than  the  primary 

primary  testing 

particular  test. 

that  are  required  for 
the  test  (e.g.,  chase 
aircraft,  software 
version  type,  etc.). 

the  test  (e.g.,  SIL,  Iron 
Bird,  computational 
fluid,  dynamics  software, 
structural  loads  software 
models,  etc.). 

the  test  (secondary  or 
related  facilities  can  be 
listed  in  parentheses). 

crew/testers  who  are 
required  to  conduct 
the  particular  test. 

facility  does  not 
possess  necessary 
to  conduct  the 

test. 
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testing  various  altitudes,  offload  quantities,  and 
lighting  conditions;  thus  minimizing  test  flights  to 
verify  and  validate  receivers. 

The  Joint  Cargo  Aircraft  program  also  is  combining 
DT  and  OT  events  to  perform  integrated  testing.  Like 
the  C-130J  ALR-56M  testing,  the  range  time  required 
to  conduct  defensive  system  testing  is  one  area 
considered  for  IT.  The  risk  of  changes  to  the  defensive 
system  had  been  considered  higher  for  this  program,  so 
inside  of  conducting  integrated  test  events  at  the  end  of 
each  DT  mission,  all  the  OT  missions  are  planned  to 
occur  during  the  last  2  weeks  of  the  defensive  system 
testing.  The  test  teams  chose  this  method  to  allow  DT 
and  the  program  office  to  make  changes  required  prior 
to  the  OT  events  but  still  reap  the  cost  and  schedule 
savings.  OT  events  will  stih  occur  when  the  aircraft  is 
already  at  the  range  instead  of  another  deployment  to 
the  ranges  at  a  later  date. 

AFOTEC  is  committed  to  an  early  influence 
process  supporting  and  enhancing  acquisition  pro¬ 
grams.  Involvement  in  the  KC-X  and  C-130J 
electronic  warfare  range  testing  and  formation  test 
programs  over  the  past  2  years  has  illustrated  the 
commitment.  Constructing  an  ITEM  to  support  IT 
development  for  the  KC-X  program  is  just  one 
example.  Aiding  system  training  requirements  refine¬ 
ment,  assisting  development  for  concept  of  employ¬ 
ment  documents  with  the  user,  and  identifying 
potential  acquisition  issues  affecting  test  are  further 
steps  AFOTEC  takes  to  aid  early  influence  efforts.  All 
of  these  early  influence  efforts  culminate  in  the  OT 
plan  with  integrated  test  events.  By  collaborating  with 
all  stakeholders  through  the  early  influence  process, 
AFOTEC  enables  delivery  of  warfighter  capabilities 
faster,  with  more  confidence,  and  meeting  the 
increased  demands  on  T&E  to  rapidly  replace  current 
airframes. 

Conclusion 

AFOTEC  action  officers  have  worked  directly  with 
developmental  testers,  planning  to  eliminate  test 
redundancy  and  capitalizing  on  data  sharing  during 
integrated  test  events.  The  efforts  have  focused  on 
synergizing  test  events  and  objectives.  Further  ITT 
meetings  will  incorporate  system  group  and  user 
logistic  representatives  as  well  as  requirements,  tactics, 
current  operations,  and  information  assurance  repre¬ 
sentatives  to  reduce  test  redundancy  and  capitalize  data 
sharing  on  similar  test  events. 

In  the  introduction  to  this  article,  we  concluded  that 
IT  requires  the  following: 

1.  collaborative  planning  and  execution  of  test 
events; 


2.  shared  data; 

3.  stakeholder  independent  data  analysis,  evalua¬ 
tion,  and  reporting; 

4.  establishing  risks  and  risk  mitigation  to  decision 
makers  in  a  timely  manner; 

5.  efficient  use  of  limited  assets,  reducing  overall 
time  required  for  testing; 

6.  elimination  of  unnecessary  overlap  of  test  events; 
and 

7.  documentation  in  an  OT&E  plan  with  integrated 
events  signed  by  the  OT  commander  and 
acknowledged  by  the  PEO. 

The  KC-X,  C-130J,  JCA  and  E-3  DRAGON 
programs,  through  the  establishment  of  their  ITT  and 
early  test  planning  efforts,  are  meeting  the  collabora¬ 
tive  planning  requirements  outlined  in  DoD  instruc¬ 
tions  and  senior  leadership  directives.  Personnel  from 
the  program  office,  user  command,  FAA,  develop¬ 
mental  and  operational  test  organizations  are  already 
collaborating  on  the  test  program.  Additionally, 
creation  of  the  E-3  DRAGON  and  KC-X  test  events 
integrated  planning  matrix  and  OTStE  plans  with 
integrated  events  support  collaborative  execution  of 
events  and  the  ability  to  share  the  data  from  those 
events,  leading  to  test  optimizations.  The  optimiza¬ 
tions  ensure  the  most  efficient  and  expedient  utiliza¬ 
tion  of  limited  assets  and  eliminate  unnecessary  test 
event  overlap.  Construction  of  the  ITEM  and  sharing 
the  document  helps  communicate  the  IT  concept, 
allowing  the  test  community  to  begin  defining  their 
test  plans  before  the  contract  is  awarded.  Including  the 
ITEM  in  the  TEMP  will  also  highlight  test  risk  areas. 

The  mobility  and  surveillance  aircraft  IT  planning 
should  yield  significant  economies,  while  reducing  the 
overall  test  footprint  and  expenditures.  The  results  can 
achieve  shorter  acquisition  timelines,  while  balancing 
cost,  performance,  and  schedule  requirements  while 
meeting  the  demand  for  air  asset  requirements  in  the 
21st  century.  □ 
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Endnotes 

1DRAGON  =  DMS  (Diminishing  Manufacturing  Sources)  Replace¬ 
ment  of  Avionics  for  Global  Operations  and  Navigation. 

2The  KC-X  acquisition  strategy  is  under  review  within  the  DoD  and 
may  change  in  the  future. 
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